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Abstract 

Many  organizations  have  turned  to  alternative  voting  sys¬ 
tems  such  as  instant-runoff  voting  for  determining  the  out¬ 
come  of  single-winner  elections.  It  is  our  position  in  this 
paper  that  the  increasing  deployment  of  such  alternative 
systems  necessitate  the  study  and  development  of  risk- 
limiting  audits  for  these  systems.  We  initiate  this  study. 

We  examine  several  commonly  used  single-winner  vot¬ 
ing  systems  and  provide  risk-limiting  auditing  procedures 
for  them.  In  many  cases  the  methods  from  auditing  plural¬ 
ity  contests  can  be  applied  with  minor  changes  and  little 
loss  in  efficiency.  For  instant-runoff  voting  (IRV),  the 
situation  is  markedly  different.  We  describe  an  algorithm 
for  auditing  the  candidate  elimination  order  using  plural¬ 
ity  methods  which  is  risk-limiting.  Standard  risk-limiting 
methods  can  be  employed  if  the  margin  of  the  election 
can  be  efficiently  calculated  or  bounded.  We  provide  effi¬ 
ciently  computable  upper  and  lower  bounds  on  the  margin 
and,  when  known,  compare  them  to  the  exact  margins. 
Both  auditing  algorithms  are  potentially  far  less  efficient 
than  the  methods  to  audit  other  types  of  voting  systems. 

1  Introduction 

A  post-election  audit  is  a  procedure  that  compares  elec¬ 
tronic  vote  tallies  with  paper  ballots  in  an  attempt  to 
determine  if  the  outcome  is  correct.  For  example  in  a 
single-winner  elections,  the  audit  is  to  decide  if  the  win¬ 
ner  is  correct,  not  the  tallies  of  the  votes.  A  risk-limiting 
audit  is  one  for  which  there  is  a  known  probability  —  the 
risk  level  —  of  certifying  the  reported  outcome  when  it 
is  incorrect.  In  this  paper  we  describe  how  to  perform 
risk-limiting  audits  for  elections  that  use  voting  systems 
other  than  plurality  or  first-past-the-post. 

California  law  requires  a  1%  manual  tally  of  each 
election  that  compares  the  paper  ballots  to  the  machine 
records.  Municipal  elections  in  San  Francisco  use  instant- 
runoff  voting  (IRV).  For  these  IRV  elections,  the  follow¬ 
ing  manual  tally  procedure  is  employed  [30].  First,  in 
each  randomly  chosen  precinct,  the  paper  ballots  are  ex¬ 
amined  to  determine  the  number  of  first-choice,  second- 
choice,  and  third-choice  votes  each  candidate  received;1 
these  totals  are  compared  against  the  corresponding  totals 


San  Francisco  allows  voters  to  rank  no  more  than  three  of  the 
candidates  for  each  race. 


claimed  in  the  original  machine  count.  Second,  an  IRV 
elimination  election  is  run  with  only  the  ballots  from  the 
tallied  precinct,  and  the  winner  of  this  mini  election  is 
noted. 

There  is  no  reason  to  believe  that  the  San  Francisco 
tally  of  IRV  elections  is  a  risk-limiting  audit  for  any  partic¬ 
ular  risk  level.  Indeed,  the  San  Francisco  Voting  Systems 
Task  Force  gives  an  example  election  in  which  two  sets 
of  ballots  that  are  identical  under  the  tally  procedure  pro¬ 
duce  two  different  election  outcomes  [30,  Appendix  A].2 
In  this  example  election,  running  San  Francisco’s  man¬ 
ual  tally  and  finding  no  discrepancies  does  not  increase 
our  confidence  that  the  reported  and  actual  winner  are 
the  same!  By  contrast,  San  Francisco’s  1%  manual  tally 
of  a  plurality  election  does  provide  a  risk-measuring  au¬ 
dit,  though  the  risk  level  depends  on  the  election  mar¬ 
gin. 

Plurality  voting  has  many  well-documented  drawbacks 
which  has  motivated  the  adoption  of  alternative  voting 
systems.  Proponents  of  these  systems  claim  that  voters 
can  more  clearly  express  their  preferences  and  that  the 
election  winners  better  reflect  the  aggregate  opinion  of 
the  electorate.  Yet  in  the  absence  of  a  risk-limiting  audit, 
it  is  difficult  to  argue  that  voters’  preferences  have  been 
correctly  aggregated  —  to  convince  the  loser  that  (s)he 
lost. 

It  is  our  position  in  this  paper  that  the  increasing  deploy¬ 
ment  of  alternative  voting  systems  such  as  IRV  requires 
the  development  of  risk-limiting  audits  for  these  alterna¬ 
tive  systems.  We  initiate  the  study  of  risk-limiting  audits 
for  a  variety  of  alternative  voting  systems,  by  adapting 
techniques  previously  used  for  auditing  plurality  elections. 
An  important  ingredient  in  developing  audits  for  these 
systems  is  to  calculate  the  margin  of  victory  and  how  er¬ 
rors  in  ballots  affect  the  margin.  In  some  cases  calculating 
the  margin  is  intuitive,  but  it  is  harder  to  understand  the 
margin  for  IRV  elections.  There  is  likely  room  for  sub¬ 
stantial  improvement  over  our  audit  techniques,  and  we 
hope  that  future  work  will  provide  these  improvements. 

In  this  paper,  we  are  concerned  only  with  ballot-based, 
risk-limiting  audits.  An  audit  is  ballot-based  if  each  ballot 
cast  in  the  election  can  be  independently  selected  and 


-In  a  presentation  at  the  EVN  2011  conference,  Emily  Shen  gave 
another  such  example. 
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compared  to  the  corresponding  paper  ballot.  A  measure 
of  an  auditing  algorithm’s  efficiency  is  the  number  of 
ballots  that  must  be  examined  in  the  case  that  the  reported 
outcome  is  correct.  A  simple  risk-limiting  audit  compares 
every  electronic  record  to  its  paper  ballot.  This  is  clearly 
inefficient  since  it  requires  examining  each  ballot. 

Background:  single-winner  voting  systems.  An  elec¬ 
tion  is  single-winner  if  exactly  one  candidate  is  declared 
the  winner  in  each  contest.  In  practice,  not  all  elections 
are  single-winner;  for  example,  the  three  candidates  with 
highest  vote  totals  in  a  contest  for  city  council  might  all 
capture  seats  on  the  council.  We  focus  on  single-winner 
elections  in  this  paper  to  simplify  the  analysis,  but  our 
techniques  might  extend  to  multiple-winner  elections. 

The  most  common  single-winner  voting  system  in 
use  is  plurality  voting  —  also  called  first-past-the-post 
or  winner-takes-all.  Well-known  deficiencies  in  plural¬ 
ity  voting  have  led  to  the  development  and  use  of  other 
single-winner  systems  such  as  approval  voting,  Condorcet 
methods  and  instant-runoff  voting  (IRV).  Arrow’s  impos¬ 
sibility  theorem  [1]  and  the  Gibbard-Satterthwaite  theo¬ 
rem  [  14,  3 1  ]  tell  us  that  no  non-dictatorial  election  method 
based  on  preference  ranking  will  simultaneously  satisfy 
several  desirable  fairness  criteria.  The  field  of  social 
choice  theory  is  full  of  “paradoxes”  of  this  nature  that 
highlight  difficulty  in  aggregating  individual  preferences 
into  a  group  preference;  more  popular  accounts  can  be 
found  in  recent  books  [25,  28,  38]. 

In  the  academic  literature  on  post-election  auditing, 
plurality  voting  has  received  the  most  attention,  with  very 
little  work  on  simple  variants  such  as  range  voting,  and 
until  very  recently,  almost  no  attention  on  elections  using 
ranked-choices  (Condorcet,  IRV).  In  this  paper  we  show 
that  in  many  cases  we  can  adapt  or  modify  approaches 
from  risk-limiting  audits  of  plurality  voting  to  other  single¬ 
winner  voting  systems.  However,  in  the  case  of  IRV,  the 
auditing  problem  is  more  difficult  and  our  approaches  are 
not  as  efficient  as  for  other  methods. 

The  apparent  difficulty  of  auditing  IRV  is  ironic.  IRV 
is  a  popular  alternative  to  plurality  voting  because  it  pur¬ 
ports  to  avoid  the  spoiler  effect  and  encourages  many 
candidates  to  run.  This  can  have  policy  benefits,  as  po¬ 
litical  parties  can  receive  funding  and  recognition  if  they 
capture  the  first  preferences  of  a  certain  portion  of  the 
electorate  [27].  These  benefits  could  be  nullified  if  it  is 
difficult  to  ascertain  that  an  IRV  election  was  correctly 
decided. 

In  this  paper,  we  focus  our  attention  on  voting  systems 
that  enjoy  some  level  of  popularity.  Other,  less  used,  vot¬ 
ing  systems  are  considered  when  the  problem  of  auditing 
such  systems  reduces  to  that  of  auditing  more  popular  sys¬ 
tems  or  when  audit  procedures  can  be  obtained  through 
minor  modifications  to  other  techniques. 


Our  contributions.  In  this  paper,  we  propose  post¬ 
election  risk-limiting  auditing  schemes  for  several  popular 
single-winner  voting  systems  and  variants. 

We  classify  the  voting  systems  we  study  into  two 
groups:  scored  systems  and  ranked  systems.  We  show 
that  the  first  group,  which  includes  approval  voting,  range 
voting,  and  Borda  counts,  can  be  audited  using  the  anal¬ 
ysis  and  algorithms  from  plurality  election  auditing.  In 
some  cases,  we  are  able  to  reduce  auditing  systems  in 
the  second  group  to  multiple-contest  plurality  elections. 
We  can  then  apply  the  method  of  Stark  [37]  to  audit  the 
constituent  plurality  contests  simultaneously  with  a  given 
total  risk  of  making  a  mistake.  In  particular,  we  give 
the  first  risk-limiting  auditing  algorithm  for  Condorcet 
methods  (when  there  is  a  Condorcet  winner)  and  for  IRV. 

2  Non-plurality  voting  and  related  work 

We  divide  election  methods  for  single-winner  contests 
into  two  classes.  These  classes  are  distinguished  by  how 
the  system  counts  a  user’s  preference  rather  than  how 
the  user  expresses  that  preference.  The  first  class,  scored 
systems ,  contains  systems  in  which  users’  preferences  are 
converted  into  a  sequence  of  numerical  scores  for  each 
candidate.  The  second  class,  ranked  systems ,  contains  sys¬ 
tems  in  which  comparisons  are  made  based  on  the  rank¬ 
ordering  of  candidates  by  the  voters.  Examples  of  the 
former  include  plurality  voting,  approval  and  range  vot¬ 
ing,  and  the  Borda  count.  Examples  of  the  latter  include 
various  Condorcet  methods  and  instant-runoff  voting. 

Scored  systems  are  perhaps  the  most  familiar  in  mod¬ 
ern  day-to-day  life.  One  might  choose  to  “like”  a  piece  of 
information  posted  on  a  social  networking  site,  implicitly 
assigning  a  score  of  1 ;  and  one  often  encounters  restau¬ 
rant  reviews  that  assign  ratings  on  a  four-star  scale.  When 
electing  candidates,  as  with  deciding  which  piece  of  in¬ 
formation  is  most  popular  or  deciding  which  restaurant 
is  best,  the  procedure  is  obvious:  add  up  all  of  the  scores 
and  pick  the  alternative  with  the  highest  score. 

Ranked  systems  are  less  common  and  the  procedure 
for  picking  a  winner  is  less  straightforward.  The  social 
choice  literature  is  replete  with  methods  of  aggregating 
preference  rankings.  Arrow’s  Theorem  shows  that  no  ag¬ 
gregation  method  can  simultaneously  satisfy  a  set  of  de¬ 
sirable  criteria,  which  means  there  is  no  “perfect”  method. 
The  reasons  for  choosing  one  method  over  another  in¬ 
volve  evaluating  tradeoffs  between  different  notions  of 
fairness  or  desiderata  exogenous  to  the  mathematics  of 
the  method. 

Approval  and  range  voting.  In  approval  voting,  voters 
simply  mark  “yes”  or  “no”  for  each  candidate  to  indicate 
whether  or  not  they  approve  of  them.  Range  voting  al¬ 
lows  voters  to  assign  a  numerical  score  (e.g.,  1  to  10)  to 
their  approval  to  indicate  the  degree  to  which  they  ap- 
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prove  of  each  candidate.  In  both  cases,  the  candidate 
with  the  highest  number  of  points  or  “yes”  votes  wins. 
Since  voters  can  approve  of  more  than  one  candidate, 
advocates  of  approval  and  range  voting  claim3  it  avoids 
the  “spoiler  effects”  rampant  in  plurality  voting,  wherein 
a  third  candidate  siphons  votes  from  one  of  the  top  two 
candidates. 

Borda  counts.  Many  of  the  classical  results  in  voting 
date  to  the  time  of  the  French  Revolution.  In  Borda  count 
elections,  voters  submit  their  preferences  in  terms  of  a 
ranking  of  the  candidates.  Jean-Charles  de  Borda  pro¬ 
posed  a  method  of  tallying  ranked  ballots  by  assigning 
points  to  each  rank  and  giving  these  points  to  the  candi¬ 
dates  [6,  15].  In  a  5  candidate  election,  a  voter  would  give 
5  points  to  their  first-ranked  candidate,  4  to  the  second- 
ranked,  3  to  the  third-ranked,  and  so  on.  Borda  counts  are 
used  for  some  political  elections  in  Slovenia  [11],  as  well 
as  the  pacific  island  nations  of  Nauru  and  Kiribati  [26],  but 
is  most  popular  in  the  worlds  of  sports  —  e.g.,  the  Heis- 
man  Trophy  [17]  —  and  academic  professional  societies. 

Condorcet  methods.  Condorcet  methods  are  a  class  of 
voting  systems  in  which  voters  rank  candidates.  For  each 
pair  of  candidates  i  and  j,  the  electorate  is  said  to  prefer 
i  to  j  if  more  ballots  rank  i  above  j  than  vice  versa.  A 
Condorcet  winner,  if  one  exists,  is  a  candidate  i  who  is 
preferred  to  each  other  candidate.  A  tabulation  method 
satisfies  the  Condorcet  criterion  if  it  elects  the  Condorcet 
winner  when  one  exists.  Any  method  that  satisfies  the 
Condorcet  criterion  is  a  Condorcet  method. 

It  is  possible  to  write  endlessly  on  Condorcet  meth¬ 
ods.  Indeed,  a  search  through  the  literature  will  turn  up 
a  wide  assortment  of  election  mechanisms  which  satisfy 
the  Condorcet  criterion  in  addition  to  a  host  of  other  crite¬ 
ria.  For  descriptions  and  discussion  of  the  most  common 
Condorcet  methods  prior  to  1980,  including  Condorcet’s 
original,  see  Fishburn  [12]  and  Tideman  [39].  In  the  latter 
work,  Tideman  also  describes  his  ranked  pairs  method 
where  each  pair  of  candidates  is  compared  on  every  ballot. 
If  there  is  a  Condorcet  winner,  she  is  elected.  Otherwise, 
the  winner  is  chosen  by  considering  the  magnitudes  of  the 
victories  in  the  pairwise  elections.  More  recently,  Markus 
Schulze  proposed  what  has  become  the  most  commonly 
used  Condorcet  method,  including  by  the  Swedish  Pi¬ 
rate  Party  for  primaries,  the  Wikimedia  Foundation,  the 
Debian  project,  and  the  Gentoo  project  [33]. 4 

Several  Condorcet  methods  have  been  designed  to  sat¬ 
isfy  additional  desirable  criteria  beyond  the  Condorcet 

3See,  for  example  http :  /  /rangevoting .  org. 

4Wikipedia  lists  60  organizations  which  use  the  Schulze  Method 
in  some  form.  http://en.wikipedia.Org/w/index.php? 
title=Schulze_method&oldid=434396935#Use_of„ 
the_Schulze_method  Accessed  201 1-06-15. 


criterion  such  as  the  Condorcet  loser  criterion,  the  Smith 
criterion  (also  called  the  generalized  Condorcet  crite¬ 
rion),  and  the  independence  of  clones  criterion.  The 
social  choice  literature  has  extensively  analyzed  these 
and  other  properties  of  voting  systems.  See  Fishburn  [12], 
Woodall  [40],  Schulze  [32,  33]  and  the  references  therein 
for  details. 

Instant-runoff  voting.  In  instant-runoff  voting  (IRV)  — 
sometimes  called  the  alternative  vote  (AV),  ranked  choice 
voting  (RCV),  or,  for  the  multiseat  version,  single  trans¬ 
ferable  vote  (STV)  —  voters  also  submit  their  preferences 
as  a  (possibly  truncated)  ranked  list  of  the  candidates.  It 
is  probably  the  most  widely  used  non-plurality  voting 
system  for  political  elections.  The  Australian  House  of 
Representatives  uses  STV  [9],  as  does  the  Republic  of 
Ireland  for  all  public  elections  including  presidential  elec¬ 
tions  and  elections  to  Dail  Eireann  —  the  lower  house  of 
parliament  [11].  The  California  cities  of  Berkeley,  Oak¬ 
land.  San  Francisco,  and  San  Leandro  use  IRV  for  some 
elections.  In  an  IRV  election,  candidates  are  eliminated 
sequentially,  beginning  with  the  candidate  receiving  the 
fewest  first-ranked  votes.  The  ballots  whose  first-ranked 
candidate  was  eliminated  are  assigned  to  their  second- 
ranked  candidates.  A  more  detailed  description  of  IRV  is 
given  in  Section  4.2. 

The  IRV  elections  in  Australia  use  paper  ballots  where 
voters  fill  out  an  explicit  ranking  for  all  of  the  candidates. 
The  ballots  are  twice  counted  by  hand  under  the  watch  of 
“scrutineers”  —  the  election  observers  [9].  In  California, 
voters  only  choose  their  top  three  choices,  no  matter  how 
many  candidates  are  on  the  ballot. 

California  law  mandates  a  1%  manual  tally  to  validate 
the  outcome  of  an  election.  For  IRV  elections,  this  in¬ 
volves  selecting  1%  of  the  precincts  at  random  and  for 
each  precinct,  counting  the  number  of  first,  second,  and 
third  choices  for  each  candidate,  and  then  running  a  “mini- 
IRV”  election  using  only  the  ballots  for  the  precinct.  This 
is  not  a  risk-limiting  procedure  and  that  data  produced 
by  it  (the  tallies  and  within-precinct  IRV  winner)  give 
no  information  about  the  election  as  a  whole  [30,  Ap¬ 
pendix  A] . 

One  particularly  challenging  problem  we  consider  is 
computing  the  margin  for  an  IRV  election.  Two  recent 
papers  consider  this  problem  as  well.  One,  by  Cary  [7], 
gives  an  algorithm  that  computes  a  lower  bound  for  the 
margin.  Another,  by  Margino  et  al.  [21],  gives  an  algo¬ 
rithm  that  for  many  real  IRV  elections  is  able  to  compute 
the  margin  exactly.  Though  these  papers  were  indepen¬ 
dent  of  and  simultaneous  with  our  initial  work,  in  this 
revised  version  we  make  use  of  Cary’s  lower  bound,  and 
(where  it  is  available)  of  Magrino  et  al.’s  exact  margin.  To 
these  we  add  an  algorithm  for  computing  an  upper  bound 
for  the  margin. 
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Risk-limiting  audits.  In  this  paper  we  are  interested  in 
the  problem  of  auditing  the  outcome  of  an  election.  An 
audit  is  a  procedure  in  which  samples  of  the  ballots  cast 
in  the  election  are  drawn  and  compared  to  the  electronic 
record  of  the  ballot  used  in  the  tally.  The  purpose  of  the 
audit  is  to  look  for  evidence  that  the  reported  winner  is  not 
actually  the  winner.  If  the  procedure  does  not  find  enough 
such  evidence,  the  reported  winner  is  declared  the  true 
winner.  An  audit  procedure  is  called  risk-limiting  at  risk 
level  a  if  the  probability  (over  the  sampling  procedure) 
that  it  incorrectly  confirms  the  reported  winner  is  less 
than  a.  Our  underlying  assumption  is  that  some  of  the 
ballots  may  be  miscounted  due  to  human  or  machine  error. 

There  has  been  a  great  deal  recent  work  on  performing 
risk-limiting  audits.  See  Checkoway  et  al.  [8],  Stark  [37], 
and  the  references  therein  for  a  thorough  discussion. 

We  define  the  margin  of  an  election  to  be  twice  the 
number  of  erroneous  ballots  needed  to  change  the  winner 
of  an  election.  In  the  case  of  plurality  elections,  this  is 
the  gap  between  the  top  two  candidates  when  the  that 
gap  is  even,  and  one  more  than  that  when  the  gap  is  odd5 
This  margin  can  be  computed  for  alternatives  to  plurality 
voting  as  well.  For  example,  consider  an  approval  vote 
election  between  two  candidates.  If  all  voters  approve 
of  the  first  candidate  but  only  half  approve  of  the  second 
candidate,  then  the  old  margin  is  n/ 2.  However,  n /2  votes 
need  to  be  changed  to  change  the  outcome  so  the  new 
margin  is  n.  For  auditing  purposes,  we  think  this  is  the 
right  definition  since  it  correctly  mirrors  the  notion  of 
detecting  an  appropriate  number  of  material  errors  in  a 
risk-limiting  audit  [36], 

An  important  consideration  for  ballot-based  audits  is 
that  they  be  efficient  in  the  sense  that  the  number  of  re¬ 
counted  ballots  required  to  certify  the  election  should 
not  be  too  large.  Many  jurisdictions  require  an  auto¬ 
matic  recount  if  the  margin  below  a  certain  percentage 
of  the  votes  cast.  For  larger  margins,  it  is  desirable  to 
develop  audits  which  require  as  few  ballots  as  possible 
to  certify  (when  the  outcome  is  correct)  while  remaining 
risk-limiting  (when  the  outcome  is  not). 

Ballot  errors  and  strategic  voting.  The  auditing  prob¬ 
lem  is  different  than  the  problem  of  strategic  voting  stud¬ 
ied  in  social  choice  theory  and  elsewhere.  Strategic  voting 
refers  to  the  problem  of  voters  casting  ballots  which  do 
not  reflect  their  true  preferences  in  the  hopes  of  exploit- 

5There  is  actually  a  slight  difference  concerning  ties  and  even  vs. 
odd  margins  which  we  will  ignore  in  what  follows.  Additionally,  when 
the  threshold  percentage  of  votes  a  candidate  needs  to  be  elected  is 
different  from  50%,  for  example  a  when  a  supermajority  is  required, 
shifting  a  vote  from  the  reported  loser  to  the  reported  winner  can  change 
the  margin  by  less  than  2  [16].  The  factor  of  2  in  the  definition  of  the 
margin  is  simply  to  agree  with  the  standard  definition  of  a  margin  for  a 
plurality  election  and  can  safely  be  ignored  in  everything  that  follows  if 
one  wishes  to  work  directly  with  number  of  changed  ballots. 


ing  the  structure  of  the  ballot  tabulation  system.  It  is 
well  known  that  many  voting  systems  are  susceptible  to 
strategic  voting  [13,  14,  31].  Determining  if  strategic 
voting  will  work  in  a  given  election  is  easy  in  the  case 
of  plurality  voting,  Borda  count,  and  Condorcet  voting 
[4],  but  Bartholdi  and  Orlin  [3]  have  shown  that  it  is 
NP-complete  for  a  voter  to  find  a  preference  order  to 
ensure  the  election  of  a  particular  candidate  under  STV. 
However,  if  the  number  of  candidates  is  small,  the  compu¬ 
tational  complexity  may  be  tolerable.  More  recent  work 
by  Conitzer  et  al.  has  studied  strategic  voting  of  many 
systems  from  the  perspective  of  computational  complex¬ 
ity  [10], 

Strategic  voting  arises  because  voters  have  an  incentive 
to  cast  ballots  that  do  not  reflect  their  true  preferences. 
However,  from  the  auditor’s  perspective,  the  voters’  true 
preferences  are  irrelevant;  a  post-election  audit  is  con¬ 
cerned  with  making  sure  that  the  voters’  expressed  pref¬ 
erences  are  counted  correctly.  A  common  question  with 
both  strategic  voting  and  auditing  is  the  following:  Given 
the  ballots  cast  in  an  election,  how  large  a  subset  must  an 
adversary  control  in  order  to  force  a  particular  outcome 
of  the  election?  From  the  perspective  of  strategic  vot¬ 
ing,  this  subset  is  a  coalition  of  strategic  voters.  From 
the  perspective  of  auditing,  the  subset  is  the  minimum 
number  of  errors  required  to  change  the  outcome  of  the 
election. 

3  Scored  systems 

Some  methods  proposed  for  auditing  elections  based  on 
plurality  voting  can  be  easily  extended  to  single-winner 
elections  in  which  voters’s  preferences  can  be  interpreted 
as  scores  given  to  each  candidate.  Efficient  risk-limiting 
audits  can  be  achieved  using  the  methods  of  Stark  [37]  or 
Checkoway  et  al.  [8].  We  illustrate  our  ideas  by  slightly 
generalizing  the  method  of  Stark  below. 

Let  k  denote  the  number  of  candidates  in  the  election 
and  let  [k]  =  {1,2,  ...,&}  denote  the  set  of  candidates. 
Let  n  be  the  total  number  of  ballots  cast  in  the  election. 
The  true  value  of  ballot  i  is  x,-  =  (x,(l),x,-(2), . . .  ,x,(£))  € 
[0, 1]*,  where  x,(j)  is  the  score  that  voter  i  gives  to  can¬ 
didate  j.  Ballot  i  is  recorded  as  y ,■  G  [0, 1  }k .  The  true  and 
reported  outcomes  are 

n  n 

p  =  5>,  Q  =  5>-  (1) 

i=i  i=i 

The  reported  winner  and  runner-up  are 

wr  =  argmaxy€[t]{e(;)},  (2) 

lr  =  argma xJe[k]{Q(j)  :  j  ^  wr},  (3) 

whereas  the  actual  winner  and  runner-up  are 
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wa  =  argmax/{P(j)},  (4) 

la  =  argma Xj{P(j)  :j^wa},  (5) 

We  will  frequently  write  w  for  wr  since  the  auditor  only 
knows  w r.  The  reported  and  actual  margins  are 


inflates  the  error)  which  helps  make  the  test  statistic  more 
robust. 

The  approach  described  above  by  Stark  [37]  was  pro¬ 
posed  in  the  context  of  auditing  plurality  contests  with 
possibly  more  than  one  winner.  It  is  easy  to  transform 
approval,  range  voting,  and  Borda  counts  into  this  frame¬ 
work,  as  we  can  show  in  the  remainder  of  this  section. 


mr  =  Q{wr)-Q(lr),  ma=P(wa)-P(la).  (6) 


Note  that  these  are  measured  in  actual  votes,  not  fractions 
or  percentages  of  the  number  of  ballots  cast. 

A  uniformly  sampled,  ballot-based  audit  consists  of 
drawing  K  ballots  uniformly  from  the  set  of  n  ballots.  Let 
Zf  =  (Xf ,  Y t)  denote  the  f-th  ballot  in  the  sample  (that  is, 
Zr  is  ballot  (xj,y,-)  for  some  i).  The  relative  overstatement 
between  candidates  w  and  j  with  respect  to  the  reported 
margin  for  the  f-th  ballot  is 


(Yt(w)-Y,(j))-(Xt(w)-Xt(j)) 

Q(w)-Q{j) 


(7) 


For  elections  where  votes  are  in  {0, 1}*,  this  is  0  when 
there  is  no  error,  positive  (either  \/{Q(w)  -Q{j))  OT 
2/(ew  —  Qij  )  ) )  when  there  is  an  error  that  decreases 
the  margin,  and  negative  (either  —  1  /\Q{w)  -  Q{j ))  or 
—2/ ( Q(xv )  —  Q(j)))  when  there  is  an  error  that  increases 
the  margin.  The  expected  value  of  et(w,j)  is 


E[ef(wJ)] 


k(QW-QU))-l;(PW-PU)) 
Q(w)  -  Q(j) 


For  the  f-th  audited  ballot,  the  worst  case  relative  over¬ 
statement  is 


et 


-  max 

i=£xv 


(Yt(xv)-Yt(j))-(Mxv)-Xt(j)) 


This  is  conservative,  as  et  may  be  positive  if  the  f-th  sam¬ 
pled  ballot  shrinks  the  relative  gap  between  the  winner 
and  any  other  candidate  j.  The  expectation  of  e,  is  not 
easy  to  calculate,  but  it  is  sufficient  to  upper  bound  it.  The 
numerator  is  at  most  2  and  the  denominator  is  at  least  mr, 
so  e,  <  2 /'//!,-.  The  test  procedure  consists  of  sampling 
ballots  and  computing  the  test  statistic 

K 


r  0*0  =  11 


f=l 


1  -  (mr/n)/ (2y) 
1  —  etmr/(2j) 


(10) 


The  election  can  be  certified  with  risk  a  if  T(K)  <  a. 
The  parameter  y  >  1  effectively  shrinks  the  margin  mr  (or 


3.1  Approval  voting 

In  approval  voting,  each  voter  can  decide  to  approve  or 
disapprove  of  each  candidate.  Therefore  the  ballots  are 
Xj  £  {0, 1}*.  The  auditing  method  was  originally  designed 
to  work  for  the  setting  where  voters  could  approve  of  up 
to  c  candidates  and  there  were  c  winners,  so  this  is  a 
simple  extension  for  approving  of  up  to  k  candidates  with 
1  winner. 

3.2  Range  voting 

In  range  voting  users  can  assign  a  score  to  each  candidate. 
These  scores  are  typically  integers,  say  from  0  to  10.  The 
winner  is  the  candidate  who  garners  the  maximum  sum 
score  from  the  voters.  For  a  range  voting  system  with 
scores  in  the  range  0  to  S  we  can  normalize  by  S  so  that 
each  ballot  is  represented  by  x,-  £  {0, 1  /S,2/S, . . . ,  1}*. 
The  auditing  algorithm  can  then  be  run  as  before.  Note 
that  for  range  voting  the  upper  bound  of  2  jmr  on  e,  may 
be  significantly  more  conservative  than  for  approval  vot¬ 
ing,  especially  if  many  voters  do  not  have  polarized  views 
about  all  of  the  candidates.  This  decreases  the  efficiency 
of  the  audit  since  it  uses  more  ballots  than  necessary. 

3.3  Borda  count 

Borda  counts  is  thought  of  as  a  voting  system  where  users 
rank  candidates.  This  is  true  in  that  users  submit  their 
preferences  in  terms  of  a  ranked  list.  However,  the  Borda 
count  converts  this  ranked  list  into  a  numerical  score  for 
each  candidate,  and  hence  can  be  audited  by  the  same 
mechanism  as  other  scored  systems.  On  a  ballot  for  an 
election  to  be  tabulated  by  a  Borda  count,  voters  rank 
candidates  in  order  of  preference.  In  an  election  with 
k  candidates,  the  Borda  count  assigns  k  s  +  1  points  to 
the  ,v-th  highest  ranked  candidate.  Thus  the  top-ranked 
candidate  for  the  voter  gets  k  points,  the  second-ranked 
candidate  gets  k—  1  points,  and  so  on.  Voters  need  not 
rank  all  candidates;  an  unranked  candidate  gets  0  points. 
Again,  by  dividing  the  number  of  points  by  k,  we  can 
represent  the  ;-th  ballot  as  x,-  £  {0, \/k,2/k,...,  I}4. 

4  Ranked  systems 

As  discussed  in  Section  2,  some  alternative  voting  sys¬ 
tems  ask  voters  to  explicitly  rank  candidates.  We  showed 
earlier  that  the  Borda  count  is  best  thought  of  as  a  scored 
system,  but  Condorcet  and  IRV  elections  use  the  ranking 
order  in  a  fundamentally  different  way. 
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Consider  an  election  with  A  candidates  and  n  ballots 
cast.  For  a  set  A  C  [A-],  let  11(A)  denote  the  set  of  all  or¬ 
dered  subsets  of  [A],  That  is,  11(A)  contains  all  ranked 
lists  of  elements  of  A.  In  a  ranked-choice  election  with  k 
candidates,  a  ballot  x,-  for  voter  i  is  an  element  of  n ( [A] ) . 
The  ballot  x,-  is  recorded  as  a  ballot  y The  election  sys¬ 
tems  we  discuss  in  this  section  all  operate  on  the  counts 
of  the  election.  For  an  S  €  11(A)  define 

n 

N(s)  =  ^(yi  =  s).  (ii) 

i=  1 

That  is,  N(S)  is  the  number  of  ballots  recorded  as  prefer¬ 
ence  ranking  S. 

Unlike  in  scored  systems,  we  cannot  create  a  common 
framework  for  tabulating  ranked  systems,  but  the  two 
methods  we  discuss  in  this  section.  Condorcet  and  IRV, 
perform  simple  arithmetic  operations  and  comparisons 
on  the  ballots  in  order  to  compute  the  outcome  of  the 
election. 

4.1  Condorcet  methods 

To  tabulate  a  Condorcet  election,  the  counts  are  converted 
into  pairwise  preferences 

C(i,j)  =  ^V(S)  •  1  (i  precedes  j  in  S).  (12) 

Sena*]) 

That  is,  C(i,j )  is  the  number  of  ballots  in  which  i  is  ranked 
higher  than  j.  If  there  exists  a  candidate  i  £  [A]  such  that 
C(i,j)  >  C(j,i)  for  all  j  /  i,  then  candidate  i  is  called 
the  Condorcet  winner.  The  Condorcet  graph  has  vertices 
which  are  the  candidates  and  the  directed  edge  from  i  to  j 
with  weight  C(i,j). 

If  there  is  a  Condorcet  winner  we  can  audit  each  edge 
connecting  the  winner  to  the  other  candidates  in  the  Con¬ 
dorcet  graph  by  considering  a  plurality  election  between 
the  two  candidates.  Certifying  the  Condorcet  winner 
will  then  certify  the  election.  Because  we  are  concerned 
with  the  winner  only,  we  need  to  simultaneously  audit 
A  —  1  pairwise  elections  and  do  not  need  to  consider  the 
(2)  —  A+  1  other  pairwise  elections.  Under  this  auditing 
procedure,  the  reported  margin  of  an  election  with  Con¬ 
dorcet  winner  wr  is 

m,  =  min  {C(wr,j)-C(j,wr)}.  (13) 

irwr 

One  way  to  audit  these  is  to  use  Stark’s  method  of  au¬ 
diting  a  collection  of  races  simultaneously  [37]  using  a 
diluted  margin  of  p  =  mr/n. 

If  there  is  no  Condorcet  winner,  then  there  is  a  ma¬ 
jority  rule  cycle  and  we  need  to  consider  the  particular 
Condorcet  completion  method  used.  There  is  a  veritable 


menagerie  of  Condorcet  completion  methods  proposed  in 
the  literature.  To  illustrate  how  auditing  applies,  we  re¬ 
strict  our  discussion  to  a  few  examples  for  which  auditing 
is  simple  to  describe. 

Two-method  systems.  A  two-method  system  elects  the 
Condorcet  winner,  if  one  exists.  If  there  is  no  Condorcet 
winner,  than  a  completely  different  method  of  tabulating 
the  ballots  is  used.  One  possible  completion  method  to 
use  when  there  is  no  Condorcet  winner,  first  described  by 
Black  [5],  uses  Borda  count  to  decide  the  winner.  Fish- 
burn  improves  on  this  by  restricting  the  Borda  counts  to 
the  Smith  set  —  the  smallest  set  of  candidates  such  that 
each  beats  all  candidates  outside  the  set  [12,  Function  Ci], 
Auditing  a  two-method  system  involves  auditing  each 
method.  If  the  reported  counts  indicate  a  Condorcet  win¬ 
ner  we  can  audit  at  risk  level  a  using  the  methods  de¬ 
scribed  above.  If  the  reported  counts  indicate  that  there 
is  no  Condorcet  winner  we  first  audit  ballots  to  assure 
that  no  Condorcet  winner  exists  at  risk  level  r\  by  simul¬ 
taneously  auditing  the  pairwise  elections  that  form  the 
majority  rule  cycle.  We  can  then  audit  the  completion 
method  (e.g.,  Borda  count)  at  risk  level  n.  We  pick  r\ 
and  r 2  such  that 

1  —  (1  —  ri)(l  —  ri)<a.  (14) 


One-method  systems.  A  one-method  system  is  a  single 
procedure  that  elects  the  Condorcet  winner  when  one  ex¬ 
ists,  and  selects  a  different  candidate  otherwise.  For  the 
same  set  of  cast  ballots,  different  one-method  systems 
may  elect  different  candidates.  If  there  is  a  reported  Con¬ 
dorcet  winner,  the  election  can  be  audited  using  either  the 
general  method  above  or  by  auditing  the  specific  method 
used.  If  there  is  no  reported  Condorcet  winner,  then  a 
secondary  audit  must  be  used  for  the  specific  method. 

The  Nanson  method  [22]  and  the  related  Baldwin 
method  [2]  work  in  rounds  with  one  or  more  candidates 
eliminated  each  round,  similar  to  instant-runoff  voting,  ex¬ 
cept  that  Borda  counts  determine  who  is  eliminated.  The 
auditing  procedure  is  very  similar  to  IRV  (Section  4.2). 
The  Schulze  method  —  the  most  commonly  used  Con¬ 
dorcet  method  —  is  more  complicated.  Developing  a  risk 
limiting  audit  for  the  Schulze  method  is  an  open  prob¬ 
lem.  However,  most  organizations  which  use  the  Schulze 
method  do  not  use  physical  ballots  or  a  voter- verifiable  pa¬ 
per  audit  trail  (VVPAT),  so  the  auditing  framework  used 
here  may  not  be  appropriate. 

4.2  Instant-runoff  voting 

In  an  IRV  election,  voters  also  express  their  preferences  as 
a  ordered  subset  of  the  candidates.  The  counting  proceeds 
in  rounds.  In  each  round,  the  candidates  with  the  fewest 
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top-choice  votes  are  eliminated.  Eliminating  a  candidate 
effectively  removes  the  candidate  from  all  ballots  in  which 
she  was  ranked,  causing  later  ranked  candidates  to  move 
up  one  spot.  A  candidate  who  is  not  eliminated  is  called 
a  continuing  candidate.  A  ballot  is  considered  exhausted 
when  all  of  the  candidates  it  ranks  have  been  eliminated. 
The  elimination  stops  when  one  candidate  has  a  majority 
of  top-choice  votes  on  the  nonexhausted  ballots. 

There  are  several  methods  for  choosing  the  candidates 
to  eliminate.  The  simplest  is  to  eliminate  the  candidate 
with  the  fewest  top-choice  votes.  This  is  the  base  IRV 
elimination  rule.  In  San  Francisco  municipal,  ranked 
choice  voting  (RCV)  elections,  multiple  candidates  can 
be  eliminated  in  a  single  round.6  We  refer  to  this  as  the 
SF  RCV  elimination  rule.  In  both  cases,  the  sum  of  the 
top-choice  votes  for  candidates  chosen  to  be  eliminated 
is  less  than  the  number  of  top-choice  votes  for  every  can¬ 
didate  who  is  not  eliminated  (except  in  the  case  of  a  tie). 
That  is,  if  E  is  an  elimination  set,  then 

n  n 

X/1  (>’,0)  £  E)  <min]T  l(}>,(1)  =c)  (15) 

1 1  c*Et! 

where  y,  (l)  is  the  top.  noneliminated  choice  on  ballot  i. 
Other  elimination  rules  —  for  example,  eliminating  all 
candidates  who  do  not  receive  a  threshold  fraction  of  the 
top-choice  votes  each  round  —  exist  but  will  not  be  dis¬ 
cussed.  These  two  counting  method  are  standard,  but  are 
provided  for  completeness  in  Algorithm  4  of  Appendix  A. 

Tabulating  the  outcome  of  an  IRV  election  produces  a 
list  8  =  (Ei, £21  •  •  ■  ,Er)  of  sets  of  eliminated  candidates 
in  the  order  in  which  they  were  eliminated.  The  set  Er  is 
the  set  of  candidates  eliminated  in  the  r-th  round.  Under 
the  base  IRV  rules,  Er  is  always  a  single  candidate  for 
r  <R,  whereas  in  the  SF  RCV  rule  Er  may  contain  many 
candidates.  In  either  case,  once  one  candidate  has  a  ma¬ 
jority,  the  final  elimination  set  Er  may  contain  multiple 
candidates. 

There  are  at  least  three  approaches  to  designing  risk- 
limiting  audits  for  IRV  elections. 

Auditing  the  taint.  Given  the  margin  of  the  IRV  elec¬ 
tion,  we  can  try  to  evaluate  how  much  each  erroneous  bal¬ 
lot  contributes  to  the  margin  —  the  so-called  taint  [35]  — 
much  as  Stark’s  method  does  for  plurality  audits.  Suppose 
we  sample  a  ballot  X,  whose  cast  vote  record  is  Yf  /  X,, 


6S.F„  Cal.,  Charter  art.  XIII,  §  13.102(e)  (Mar.  2002),  “If  the 
total  number  of  votes  of  the  two  or  more  candidates  credited  with  the 
lowest  number  of  votes  is  less  than  the  number  of  votes  credited  to  the 
candidate  with  the  next  highest  number  of  votes,  those  candidates  with 
the  lowest  number  of  votes  shall  be  eliminated  simultaneously  and  their 
votes  transferred  to  the  next-ranked  continuing  candidate  on  each  ballot 
in  a  single  counting  operation.” 


for  example,  the  ballot  may  have  been  misinterpreted  by 
the  voting  system  or  an  adversary  may  have  caused  the 
cast  vote  record  to  be  changed  or  recorded  incorrectly.  If 
replacing  Y,  by  X,  does  not  change  the  (plurality)  mar¬ 
gin  in  any  elimination  decision  of  the  IRV  tabulation 
process,  we  can  declare  the  ballot  nonmaterial.  If  any 
margin  changes,  then  we  declare  the  ballot  error  material ; 
however,  it  is  not  clear  how  much  the  ballot  changes  the 
margin  without  recomputing  the  exact  margin  with  the 
corrected  ballot  —  a  potentially  lengthy  computation.  In¬ 
deed,  it  may  be  that  the  effect  of  a  collection  of  errors 
on  the  margin  may  not  be  the  sum  of  the  effects  of  each 
error.  Determining  how  to  efficiently  compute  the  size  of 
the  change  is  an  open  problem,  the  solution  to  which  may 
provide  drastically  more  efficient  auditing  methods  than 
the  two  approaches  below. 

Auditing  the  elimination  order.  A  second  approach  is 
to  audit  the  elimination  order  8  to  verify  that  the  set  of 
candidates  eliminated  in  each  round  is  correct.  If  any 
elimination  selection  is  a  result  of  a  tie  breaker  —  for 
example,  with  the  base  IRV  elimination  rule,  if  the  two 
candidates  with  the  fewest  number  of  top-choice  votes  in 
a  round  have  the  same  number  of  votes,  then  the  candidate 
to  be  eliminated  may  be  chosen  by  some  other  mecha¬ 
nism  such  as  a  coin  flip  —  then  a  complete  hand  count  is 
necessary. 

Otherwise,  each  round  of  the  algorithm  leads  to  a  plu¬ 
rality  election  to  be  audited.  For  each  round  r.  (1)  elimi¬ 
nate  and  distribute  the  votes  for  candidates  eliminated  in 
previous  rounds,  namely  £]  UGU--  -  U Er  \ ;  (2)  aggre¬ 
gate  the  candidates  who  are  to  be  eliminated  in  round  r, 
namely  those  in  Er,  into  a  “super  candidate”;  and  (3)  au¬ 
dit  a  [k!  —  1  [-winner  plurality  election  with  k!  candidates 
consisting  of  the  super  candidate  and  the  k!  —  1  continu¬ 
ing  candidates.  The  audit  in  step  (3)  is  to  ensure  that  the 
super  candidate  lost.  This  procedure  results  in  R  plurality 
elections  to  audit. 

The  R  plurality  elections  can  be  audited  simultaneously 
using  Stark’s  method  [37].  Each  ballot  can  cause  0.  1. 
or  2  errors  for  each  of  the  /(-plurality  elections;  however, 
due  to  the  nature  of  the  diluted  margin  in  Stark’s  method, 
we  take  the  maximum  of  the  errors  caused  in  any  race  as 
the  error  contributed  by  the  ballot. 

In  theory,  this  auditing  method  solves  the  problem  of 
performing  a  risk-limiting  audit  for  IRV  elections,  but 
in  practice  it  may  require  counting  too  many  ballots  us¬ 
ing  the  base  IRV  rules.  This  is  because  candidates  who 
are  eliminated  early  often  constitute  a  very  small  frac¬ 
tion  of  the  total  ballots.  For  example,  in  the  2010  Oak¬ 
land  Mayoral  election,  three  candidates  each  received 
less  than  1%  of  the  votes.  This  led  to  a  small  margin 
of  83  votes  in  round  3  out  of  a  total  of  122,264  ballots 
cast  in  the  election.  Small  pairwise  margins  for  candi- 
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dates  eliminated  early-on  in  the  counting  requires  large 
sample  sizes  to  detect  an  error  in  the  elimination  order. 
If  instead  of  the  base  elimination  rule,  the  SF  RCV  rule 
is  used,  then  8  of  1 1  candidates  are  eliminated  in  the 
first  round  and  the  smallest  margin  used  for  the  audit 
is  1,627.  We  will  return  to  this  example  several  times  in 
Section  5.2. 

Auditing  by  error  detection.  A  third  approach  to  build¬ 
ing  a  risk-limiting  audit  is  to  attempt  error  detection.  That 
is,  the  auditor  can  sample  K  of  ballots  and  compare  each 
paper  ballot  to  its  cast  vote  record  (CVR).  If  the  number 
of  ballots  with  any  error  exceeds  a  specified  threshold, 
then  a  manual  count  of  the  entire  election  is  required. 
Suppose  that  the  margin  is  m.  The  effect  of  auditing  for 
material  errors  is  to  audit  a  fictitious  plurality  contest  be¬ 
tween  two  candidates  whose  margin  is  m.  Each  material 
error  that  is  found  reduces  the  margin  by  two.  Therefore 
any  method  for  auditing  plurality  contests  may  be  adapted 
for  the  purposes  of  error  detection.  Such  an  audit  can  be 
performed  using  any  of  the  standard  methods  [19,  23,  29]. 
If  fewer  erroneous  CVRs  are  found  than  the  threshold, 
the  auditor  certifies  the  winner  of  the  election.  We  choose 
the  threshold  so  that  the  sample-size  is  risk-limiting. 

This  last  approach  to  risk-limiting  audits  requires  com¬ 
puting  the  margin  of  an  IRV  election,  which  is  a  topic  of 
recent  interest  [7,  21].  Once  the  margin  or  a  lower  bound 
on  it  is  known,  then  we  can  set  the  threshold  to  guaran¬ 
tee  a  risk-limiting  audit.  Recent  work  by  Magrino  et  al. 
calculates  the  IRV  margin  exactly  for  some  elections  [21]. 
However,  this  exact  calculation  can  be  computationally 
very  expensive,  even  with  clever  heuristics. 

5  The  margin  of  an  IRV  election 

In  this  section  we  investigate  the  problem  of  computing 
the  margin  for  an  IRV  election.  We  first  describe  some 
real  IRV  elections  and  their  features.  We  then  show  a 
lower  bound  on  the  margin  based  on  picking  elimination 
sets  in  each  round  in  such  a  way  as  to  maximize  the  differ¬ 
ence  in  votes  between  the  “super  candidate”  described  in 
Section  4.2  and  the  continuing  candidate  with  the  fewest 
votes.  In  order  to  evaluate  how  good  this  lower  bound 
is,  we  develop  an  algorithm  that  constructs  a  set  of  ballot 
errors  that  can  alter  the  winner  of  an  IRV  election.  This 
gives  an  upper  bound  which  is  often  close  to  the  lower 
bound  in  real  elections.  Our  bounds  are  fast  to  compute, 
and  when  possible  we  compare  our  bounds  to  the  exact 
margins  reported  by  Magrino  et  al.  [21]. 

Appendix  B  contains  several  toy  examples  showing 
unintuitive  aspects  of  IRV  elections,  namely  that  just  a 
few  errors  recording  ballots  for  losing  candidates  can 
dramatically  change  the  outcome  of  the  election  and  that 
even  when  IRV  elects  the  Condorcet  winner,  the  IRV 
margin  can  be  significantly  smaller. 


5.1  Margins  for  real  elections 

We  purchased  CVR  data  for  six  different  elections  that 
were  conducted  using  ranked-choice  ballots  from  Open- 
STV7.  The  three  2002  Dail  Eireann  elections  —  Dublin 
North.  Dublin  West,  and  Meath  —  are  multiple  winner 
STV  elections  which  we  include  to  stress  our  algorithms, 
not  because  they  are  representative  of  IRV  elections.  The 
others  are  IRV  elections.  CVR  data  for  an  additional  26 
San  Francisco  Bay  Area  and  Pierce  County  RCV  elections 
were  also  collected  from  the  corresponding  municipalities’ 
websites.  A  summary  of  the  data  is  given  in  Table  1. 

The  three  Dail  Eireann,  two  Burlington  mayoral,  and 
Takoma  Park  City  Council  special  elections  allowed  vot¬ 
ers  to  provide  a  complete  ranking  of  all  of  the  candidates 
on  the  ballot.  The  last  three  additionally  allowed  write-in 
candidates  although  in  the  case  of  the  two  Burlington 
mayoral  elections,  the  write-in  took  the  place  of  one  of 
the  candidates  in  the  ranking.  All  of  the  California  and 
Washington  elections  used  ballots  where  voters  pick  their 
top  three  candidates,  including  write-ins. 

One  common  feature  of  all  of  these  elections  is  that 
they  involve  a  relatively  small  number  of  ballots  compared 
to  state  and  national  elections.  As  an  extreme  example, 
only  204  people  voted  in  the  election  for  the  Takoma  Park 
City  Council.  In  such  cases  a  full  hand  count  is  easy,  and 
would  certainly  be  risk  limiting.  Indeed,  in  Australia,  IRV 
elections  are  counted  by  hand  twice  under  the  supervision 
of  scrutineers  [9].  After  tabulating  the  results  from  these 
elections  we  were  surprised  to  note  that  they  share  a  more 
interesting  common  feature:  The  winner  according  to  the 
IRV  count  was  also  a  Condorcet  winner  for  the  election 
in  every  case  except  for  the  2009  Burlington  mayoral 
election.  In  Appendix  B.2  we  show  that  the  IRV  margin 
may  be  smaller  than  the  Condorcet  margin,  even  when 
both  methods  elect  the  same  candidate. 

Computing  the  IRV  margins  exactly  can  be  a  computa¬ 
tionally  difficult  task  for  real  elections  that  contain  large 
numbers  of  candidates  or  allow  voters  to  rank  many  can¬ 
didate  on  the  ballot  [21].  Therefore,  in  the  rest  of  this 
section  we  present  lower  and  upper  bounds  on  the  mar¬ 
gin  and  examine  the  bounds  for  the  32  elections.  Where 
known  (from  [21]),  we  compare  the  bounds  with  the  exact 
margin. 

5.2  Lower  bounds  on  the  margin 

One  obvious  lower  bound  on  the  margin  in  an  IRV  elec¬ 
tion  is  the  difference  in  votes  between  the  two  candidates 
with  the  fewest  top-choice  votes  in  each  round.  Certainly 
if  the  number  of  ballots  which  are  modified  is  not  enough 
change  any  elimination  decision,  then  the  outcome  must 
be  correct.  However,  it  is  trivial  to  show  that  this  lower 
bound  is  arbitrarily  bad  by  considering  an  election  in 
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Table  1:  Election  data. 


Election 

Candidates 

Ranks 

Ballots 

Condorcet  winner 

2002  Dail  Eireann,  Dublin  North  ' 

12 

12 

43,942 

/ 

2002  Dail  Eireann,  Dublin  West' 

9 

9 

29,988 

/ 

2002  Dail  Eireann,  Meath  ' 

14 

14 

64,081 

/ 

2006  Burlington  Mayor 

6* 

5 

9,865 

/ 

2007  San  Francisco  Mayor 

18 

3 

149,465 

/ 

2007  Takoma  Park  City  Council  special.  Ward  5 

4* 

4 

204 

/ 

2008  Pierce  County  Assessor 

7* 

3 

312,771 

/ 

2008  Pierce  County  City  Council,  Dist.  2 

4* 

3 

43,661 

/ 

2008  Pierce  County  Executive 

5* 

3 

312,771 

/ 

2009  Burlington  Mayor 

6* 

5 

8,984 

2009  Pierce  County  Auditor 

4* 

3 

159,987 

/ 

2010  Berkeley  Auditor 

2* 

3 

45,986 

/ 

2010  Berkeley  City  Council,  Dist.  1 

5* 

3 

6,426 

/ 

2010  Berkeley  City  Council,  Dist.  4 

5* 

3 

5,708 

/ 

2010  Berkeley  City  Council,  Dist.  7 

4* 

3 

4,862 

/ 

2010  Berkeley  City  Council,  Dist.  8 

4* 

3 

5,333 

/ 

2010  Oakland  Mayor 

lit 

3 

122,268 

/ 

2010  Oakland  Auditor 

3* 

3 

122,268 

/ 

2010  Oakland  City  Council,  Dist.  2 

3* 

3 

15,243 

/ 

2010  Oakland  City  Council,  Dist.  4 

8* 

3 

23,884 

/ 

2010  Oakland  City  Council,  Dist.  6 

4* 

3 

14,040 

/ 

2010  Oakland  School  Board  Director,  Dist.  2 

2* 

3 

15,243 

/ 

2010  Oakland  School  Board  Director,  Dist.  4 

3* 

3 

23,884 

/ 

2010  Oakland  School  Board  Director,  Dist.  6 

2* 

3 

14,040 

/ 

2010  San  Francisco  Board  of  Supervisors,  Dist.  2 

7* 

3 

28,911 

/ 

2010  San  Francisco  Board  of  Supervisors,  Dist.  6 

15* 

3 

25,057 

/ 

2010  San  Francisco  Board  of  Supervisors,  Dist.  8 

5* 

3 

38,551 

/ 

2010  San  Francisco  Board  of  Supervisors,  Dist.  10 

22* 

3 

20,550 

/ 

2010  San  Leandro  Mayor 

7* 

3 

23,494 

/ 

2010  San  Leandro  City  Council,  Dist.  1 

4* 

3 

23,494 

/ 

2010  San  Leandro  City  Council,  Dist.  3 

2* 

3 

23,494 

/ 

2010  San  Leandro  City  Council,  Dist.  5 

3* 

3 

23,494 

/ 

*  These  are  multiseat  STV  elections  that  have  been  treated  as  IRV. 

*  Includes  a  single  combined  write-in  candidate. 

The  Ranks  column  denotes  how  many  candidates  a  voter  was  allowed  to  rank  on  the  ballot. 
There  is  a  ■/  in  the  Condorcet  winner  column  if  the  IRV  procedure  elects  the  Condorcet  winner. 
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Table  2:  Margin  bounds  from  real  elections  using  ranked-choice  ballots. 


Lower  bound 

Exact 

Upper  bound 

Condorcet 

Election 

margin 

% 

margin 

% 

margin 

% 

margin 

% 

2002  Dail  Eireann,  Dublin  North 

203 

0.5 

— 

— 

2,724 

6.2 

2,723 

6.2 

2002  Dail  Eireann,  Dublin  West 

8 

0.0 

— 

— 

1,444 

4.8 

1,443 

4.8 

2002  Dail  Eireann,  Meath 

129 

0.2 

— 

— 

6,198 

9.7 

6,197 

9.7 

2006  Burlington  Mayor 

482 

4.9 

776 

7.9 

778 

7.9 

776 

7.9 

2007  San  Francisco  Mayor 

68,060 

45.5 

— 

— 

101,676 

68.0 

101,674 

68.0 

2007  Takoma  Park  City  Council  special.  Ward  5 

36 

17.6 

— 

— 

38 

18.6 

36 

17.6 

2008  Pierce  County  Assessor 

276 

0.1 

2,222 

0.7 

2,222 

0.7 

2,221 

0.7 

2008  Pierce  County  City  Council,  Dist.  2 

4,014 

9.2 

4,014 

9.2 

4,016 

9.2 

4,014 

9.2 

2008  Pierce  County  Executive 

4,054 

1.3 

4,054 

1.3 

4,056 

1.3 

4,054 

1.3 

2009  Burlington  Mayor 

253 

2.8 

— 

— 

254 

2.8 

588 

6.5 

2009  Pierce  County  Auditor 

16,792 

10.5 

16,792 

10.5 

16,794 

10.5 

16,792 

10.5 

2010  Berkeley  Auditor 

30,711 

66.8 

30,712 

66.8 

30,712 

66.8 

30,711 

66.8 

2010  Berkeley  City  Council,  Dist.  1 

1,770 

27.5 

2,348 

36.5 

2,350 

36.6 

2,348 

36.5 

2010  Berkeley  City  Council,  Dist.  4 

777 

13.6 

1,034 

18.1 

1,034 

18.1 

1,033 

18.1 

2010  Berkeley  City  Council,  Dist.  7 

728 

15.0 

728 

15.0 

730 

15.0 

728 

15.0 

2010  Berkeley  City  Council,  Dist.  8 

1,011 

19.0 

1,756 

32.9 

1,758 

33.0 

1,756 

32.9 

2010  Oakland  Mayor 

2,025 

1.7 

2,026 

1.7 

2,026 

1.7 

2,025 

1.7 

2010  Oakland  Auditor 

33,045 

27.0 

34,162 

27.9 

34,164 

27.9 

34,162 

27.9 

2010  Oakland  City  Council,  Dist.  2 

4,349 

28.5 

4,350 

28.5 

4,350 

28.5 

4,349 

28.5 

2010  Oakland  City  Council,  Dist.  4 

131 

0.5 

4,658 

19.5 

4,658 

19.5 

4,657 

19.5 

2010  Oakland  City  Council,  Dist.  6 

3,653 

26.0 

5,206 

37.1 

5,206 

37.1 

5,205 

37.1 

2010  Oakland  School  Board  Director,  Dist.  2 

9,660 

63.4 

9,660 

63.4 

9,662 

63.4 

9,660 

63.4 

2010  Oakland  School  Board  Director,  Dist.  4 

7,089 

29.7 

7,240 

30.3 

7,240 

30.3 

7,239 

30.3 

2010  Oakland  School  Board  Director,  Dist.  6 

9,651 

68.7 

9,652 

68.7 

9,652 

68.7 

9,651 

68.7 

2010  San  Francisco  Board  of  Supervisors,  Dist.  2 

258 

0.9 

— 

— 

260 

0.9 

258 

0.9 

2010  San  Francisco  Board  of  Supervisors,  Dist.  6 

1 

0.0 

— 

— 

1,338 

5.3 

1,337 

5.3 

2010  San  Francisco  Board  of  Supervisors,  Dist.  8 

3,552 

9.2 

— 

— 

3,554 

9.2 

3,552 

9.2 

2010  San  Francisco  Board  of  Supervisors,  Dist.  10 

2 

0.0 

— 

— 

306 

1.5 

286 

1.4 

2010  San  Leandro  Mayor 

232 

1.0 

232 

1.0 

234 

1.0 

232 

1.0 

2010  San  Leandro  City  Council,  Dist.  1 

6,262 

26.7 

6,262 

26.7 

6,264 

26.7 

6,262 

26.7 

2010  San  Leandro  City  Council,  Dist.  3 

16,675 

71.0 

16,676 

71.0 

16,676 

71.0 

16,675 

71.0 

2010  San  Leandro  City  Council,  Dist.  5 

1,484 

6.3 

1,484 

6.3 

1,486 

6.3 

1,484 

6.3 

The  exact  margins  are  taken  from  Magrino  et  al.  [21]. 
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Algorithm  1:  Lower  bound  on  IRV  margin 
inputs:  candidates  A,  ballots  B 
outputs:  lower  bound  on  margin  lb 

Enqueue  (°°,  (A,B)) 

while  true  do 

p,  ( A,B )  <—  Dequeue  ( ) 

if  |A|  =  1  then 
lb  G-  p 

return 

ESets  -G-  ValidEliminationSets  ( B ) 

foreach  E  G  ESets  do 
A'^A\E 

p'  G-  minjp,  min£  l(y,-(l)  =  c)  -  £l(y,-(l)  €  £) } 
t  ceA'(=i  i=  i  i 

B'  G-  EliminateCandidates ( B ,  E) 

Enqueue  ip',  (A',B')) 


which  there  are  two  candidates  who  each  receive  exactly 
one  vote. 

The  example  of  the  2010  Oakland  Mayoral  race  men¬ 
tioned  in  Section  4.2  shows  that  different  choices  of  elimi¬ 
nation  sets  can  lead  to  different  margins  in  the  constructed 
plurality  elections  used  for  the  audit.  Instead  of  auditing 
the  elimination  order  by  considering  the  actual  elimina¬ 
tion  sets  dictated  by  the  election  rales  as  described  in 
Section  4.2,  one  can  choose  different,  valid  elimination 
sets  for  each  round  in  such  a  way  as  to  maximize  the 
margin  of  the  constructed  plurality  elections.  This  also 
provides  a  lower  bound  on  the  IRV  margin  that  can  be 
used  to  audit  the  taint  or  by  error  detection.  Considering 
the  2010  Oakland  Mayoral  election  again,  if  the  7  lowest 
candidates  are  eliminated  in  the  first  round  instead  of  the 
8  chosen  by  the  SF  RCV  rule,  then  the  smallest  margin 
used  in  the  audit  rises  to  2025. 

The  idea  to  pick  the  best  elimination  sets  to  use  comes 
directly  from  David  Cary’s  IRV  lower  bound  computation 
in  simultaneous,  independent  work  [7].  For  concreteness, 
we  describe  the  lower  bound  algorithm  and  its  correct¬ 
ness  below  but  refer  the  interested  reader  to  Cary’s  work 
for  a  more  complete  treatment  as  well  as  an  alternative 
implementation. 

The  obvious  lower  bound  is  twice  the  number  of  ballots 
necessary  to  change  the  order  that  candidates  are  elimi¬ 
nated.  However,  by  definition  of  the  elimination  set,  in 
any  round,  any  valid  choice  of  elimination  set  can  be  cho¬ 
sen  and  those  candidates  eliminated  without  changing  the 
ultimate  winner  of  the  election  (this  is  the  basis  of  the 
SF  RCV  elimination  rule).  This  is  a  relaxation  on  the  or¬ 
der  in  which  candidates  must  be  eliminated  to  ensure  the 
correct  outcome.  Consider  the  Oakland  Mayoral  election 
one  final  time.  Since  the  8  candidates  with  the  fewest  top- 
choice  votes  in  the  first  round  can  be  eliminated  without 


changing  the  winner,  it  is  immaterial  in  which  order  those 
candidates  are  eliminated  so  long  as  they  are  eliminated 
before  any  others. 

Any  sequence  of  valid  elimination  sets  thus  gives 
a  lower  bound  on  the  margin  as  follows.  If  £  = 
(E\,E2,...,Er)  is  a  valid  sequence  of  elimination  sets, 
then 

n  n 

lbs  =  min  min  V 1  (y,-  ( 1 )  =  c)  —  V 1  (yt  ( 1 )  G  E )  (16) 

i=  1  i=  1 

is  a  lower  bound  (cf.  (15)).  If  each  Er  G  £  consists  of  a 
single  candidate,  then  lbs  is  the  obvious  lower  bound. 

Since  each  valid  lb?  is  a  lower  bound,  we  can  take  the 
maximum  over  all  valid  £  to  arrive  at  the  bound 

lb  =  max  Ibg.  (17) 

valid  £ 

The  bound  lb  can  be  efficiently  computed  by  using  a 
priority  queue.  The  queue  initially  contains  the  set  of  bal¬ 
lots  with  an  infinite  priority.  The  main  loop  removes  the 
set  of  ballots  with  the  highest  priority  p.  If  all  but  one  can¬ 
didates  have  been  eliminated,  then  the  priority  is  returned 
as  the  lower  bound.  Otherwise,  all  valid  elimination  sets 
are  computed.  For  each  valid  elimination  set,  a  copy  of 
the  ballots  is  constructed,  the  candidates  in  the  set  are 
eliminated,  and  the  new  ballots  are  placed  into  the  queue 
with  priority  //  where  //  is  the  minimum  of  p  and  the 
difference  in  votes  between  the  sum  of  top-choice  votes 
for  candidates  in  the  elimination  set  and  the  top-choice 
votes  for  the  continuing  candidate  with  the  fewest  votes. 
This  procedure  is  given  in  Algorithm  1.  Since  we  are 
using  a  priority  queue,  once  we  reach  a  set  of  ballots  for 
which  all  candidates  but  one  have  been  eliminated,  every 
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other  sequence  of  elimination  sets  leads  to  a  lower  bound 
that  is  no  better. 

Note  that  the  sequence  of  elimination  sets  used  to  con¬ 
struct  the  lower  bound  in  Algorithm  1  is  the  optimal  set  to 
use  when  performing  an  audit  of  an  IRV  election  by  con¬ 
sidering  plurality  elections  for  each  round  as  described  in 
Section  4.2.  It  is  trivial  to  modify  Algorithm  1  to  return 
the  sequence  of  eliminations  used. 

The  weakness  in  the  lower  bound  is  that  it  considers  the 
slimmest  margin  in  any  elimination  decision.  However, 
the  margin  between  two  candidates  in  a  given  round  can 
be  quite  low  but  the  candidates  together  have  too  many 
votes  to  be  grouped  into  an  elimination  set.  For  example, 
in  the  2010  Oakland  City  Council,  District  4  election,  in 
round  5  — using  the  base  IRV  elimination  rule  —  Melanie 
Helby  had  3,017  top-choice  votes  and  Daniel  Afford  had 
2,886.  Neither  can  be  eliminated  in  an  earlier  round  using 
a  larger  elimination  set  and  they  cannot  be  eliminated 
together  leading  to  a  lower  bound  of  133  which  is  about 
0.5%  of  the  total  number  of  ballots  cast  in  the  election. 
By  contrast,  the  exact  margin  is  5,658  or  about  19.5%  of 
the  ballots  cast  [21], 

5.3  Algorithmic  upper  bound  on  the  margin 

In  this  section  we  develop  an  algorithm  that  takes  a  set  of 
CVRs  and  constructs  a  set  of  ballot  errors  that  changes 
the  winner  of  the  IRV  election.  This  gives  an  upper  bound 
on  the  margin  of  the  election.  This  upper  bound,  which 
is  efficiently  computable,  is  useful  to  bound  how  far  the 
lower  bound  described  in  the  previous  section  is  from  the 
exact  value  if  the  exact  value  is  not  known.  As  we  will 
show  shortly,  in  many  real  elections  the  bound  computed 
by  this  algorithm  agrees  with  the  exact  margin. 

Our  method.  Algorithm  2,  is  based  on  calculating  an 
upper  bound  on  the  margin  for  each  possible  alternative 
winner  of  the  election.  For  a  given  alternative  winner  j, 
we  calculate  a  sufficient  number  of  errors  required  to 
make  j  the  winner  of  the  election.  Because  our  algorithm 
is  “greedy”  in  a  sense,  the  total  number  of  errors  we 
calculate  may  be  quite  a  bit  larger  than  the  minimum 
number  of  material  errors  needed  to  change  the  outcome. 

The  algorithm  proceeds  as  follows:  For  an  alternative  j, 
we  tabulate  the  IRV  election  round  by  round  until  j  ap¬ 
pears  in  the  elimination  set  E.  Let  k  be  the  continuing 
candidate  with  the  fewest  top-choice  votes;  let  ,v  be  the 
number  of  top-choice  votes  for  candidates  in  E\  and  let 
m  be  the  difference  between  the  number  of  top-choice 
votes  for  k  and  .v.  The  goal  is  to  change  enough  votes 
from  continuing  candidates  to  j  to  remove  j  from  the 
elimination  set.  There  are  two  cases.  If  E  is  the  singleton 
containing  j  —  so  that  s  =  m  —  then  shifting  m/2  +  1 
votes  from  k  to  j  will  give  j  enough  votes  to  not  be  elimi¬ 
nated  in  this  round.  Otherwise,  there  are  multiple  candi¬ 
dates  in  E  and  shifting  [(s  —  1)/2J  +  1  votes  from  k  to  j 


is  enough  to  remove  j  from  the  elimination  set  due  to 
the  strict  inequality  in  (15).  Finally,  we  tabulate  the  IRV 
election  with  the  modified  votes  and  repeat  the  margin 
modification  process  until  a  different  candidate  is  elected. 
Note  that  it  need  not  be  j  who  is  elected,  since  it  is  suf¬ 
ficient  that  any  candidate  other  than  the  reported  winner 
win. 

The  greedy  part  of  the  algorithm  comes  from  how  we 
choose  the  ballots  for  k  that  are  switched  to  ballots  for  j. 
The  selection  heuristic  is  given  in  Algorithm  3.  The 
method  changes  ballots  of  the  form  (k,...)  to  ballots  with 
first-choice  equal  to  j.  We  can  write  the  elimination  or¬ 
der  £  =  (Ei  ,E2,  . . .  ,Er).  The  intuition  is  that  for  j  to 
win,  she  must  defeat  the  candidates  in  each  of  the  sets 
E i  \  {j},E2,Ei, . . .  ,Er  and  finally  vvy.  So  the  heuristic  is 
to  preferentially  change  ballots  closest  to  the  elimination 
order  of  the  election.  This  corresponds  to  lexicographi¬ 
cally  ordering  the  cast  vote  records  as  a  function  of  £  (see 
the  definition  of  a  in  Algorithm  3). 

There  are  other  heuristics  possible  for  selecting  which 
ballots  to  shift.  For  example,  since  j  must  eventually 
defeat  vvy,  it  may  be  better  to  change  preferentially  ballots 
of  the  form  (. k,wr , . . .),  thereby  greedily  reducing  the  mar¬ 
gin  between  j  and  vvy.  Another  set  of  heuristics  can  be 
derived  by  looking  at  the  Condorcet  graph  of  the  ballots 
in  round  r  and  greedily  ordering  the  ballots  to  be  changed 
by  the  Condorcet  margin.  Since  any  heuristic  generating 
a  set  of  errors  that  alter  the  outcome  of  the  election  is 
a  valid  upper  bound  on  the  margin,  we  could  take  the 
minimum  of  margins  generated  by  Algorithm  2  with  each 
ballot  ordering. 

5.4  Margin  calculations  for  real  elections 

Table  2  shows  the  results  of  margin  calculations  for  the 
32  elections  in  Table  1 .  We  show  four  margin  calcula¬ 
tions:  the  lower  bound  of  Section  5.2,  exact  margins  from 
Magrino  et  al.  [21]  (when  possible),  the  upper  bound  of 
5.3,  and  the  margin  corresponding  to  treating  the  election 
as  a  Condorcet  election. 

In  some  elections,  the  lower  bound  produces  a  margin 
which  is  less  than  0.5%,  which  is  the  threshold  for  a 
recount8  in  many  jurisdictions.  Because  the  three  2002 
Dail  Eireann  elections  were  for  multiple-winner  STV 
elections,  the  small  values  for  the  lower  bound  may  not 
be  representative.  However,  for  two  of  the  San  Francisco 
Board  of  Supervisors  elections,  the  lower  bound  produced 
a  margin  that  is  essentially  zero  whereas  the  upper  bound 
is  a  significant  fraction  of  the  number  of  ballots  cast. 

For  those  elections  where  the  exact  margin  was  calcu¬ 
lated  by  [21],  our  upper  bound  is  either  exactly  the  same 
or  within  two  ballots  of  the  exact  margin.  The  difference 
in  two  ballots  is  because  we  take  the  margin  to  be  twice 


8Cf.,  Ala.  Code  §17-16-20  (2010)  or  Fla.  Stat.  §102.141  (2010). 
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Algorithm  2:  Greedy  upper  bound  on  IRV  margin 
inputs:  candidates  A,  ballots  B  =  {y,  €  n(A)}"=1 
outputs:  upper  bound  on  margin  m 

Winner,  ElimOrder, _ IRV  (A,  \A\,B) 
foreach  j  e  A  \  {Winner}  do 

(Winner', A', B',  ElimOrder')  -f-  (Winner, A, B,  ElimOrder) 

ej  =  0 

while  Winner'  =  Winner  do 

l  «—  index  of  E  in  ElimOrder'  such  that  j  €  E 

_,_,B' •*—  IRV  (A',  l  —1,  B')  (eliminate  candidates  who  would  be  eliminated  before  j) 
foreach  £  e  ElimOrder'(l  :  l—  1)  do 

A'  <r-  A'  \  E  (  remove  eliminated  candidates ) 
k  the  candidate  in  A'  \  ElimOrder'(/)  with  the  fewest  top-choice  votes 

s  EceElimOrder'(;)(toP-choice  votes  for  c) 
m  4—  (top-choice  votes  for  k)  —  s 
if  |ElimOrder'(/)|  >  1  then 

m  4—  m  —  1  ( modifying  the  margin  by  exactly  m  is  enough  to  change  the  elimination  set ) 
B',e  <—  ModifyMargin  (B1 ,  m,  j,  k,  ElimOrder'(/ :  end),  Winner) 
ej  ej  +  e 

Winner',  ElimOrder', _  4—  IRV  (A',  |A'|,  B') 
m  4—  2 min ,{e,} 


Algorithm  3:  ModifyMargin  -  choosing  errors  to  decrease  the  margin 

inputs:  ballots  B  =  {y,  €  n(A)}"=1,  margin  m,  recipient  j,  victim  k,  ElimOrder,  Winner 
outputs:  modified  {y,}  such  that  the  margin  between  j  and  k  has  decreased  by  more  than  m,  number  of  ballots 
changed  c 

c  4—  |_m/2j  +  1 

<j  i —  (ElimOrder(2  :  end)  ||  Winner  |j  ElimOrder(l)  \  {j}  ||  j)  (  a  is  an  ordered  list ) 

(J  (k)  ||  cr\  {£}  ( Move  k  to  beginning  of  a) 

Sort  {y,}  lexicographically  according  to  a,  where  longer  matches  appear  before  shorter  matches;  e.g., 

(cr(l), <t(2), «t(3))  precedes  (cr(l), cr(2)) 

Change  the  first  c  of  the  sorted  y,  into  votes  with  j  as  the  only  choice 


the  number  of  ballot  errors  necessary  for  a  different  can¬ 
didate  to  win,  whereas  [21]  considers  the  number  needed 
to  tie.  The  tightness  of  our  upper  bound  suggests  that  the 
heuristic  that  we  used  is  an  good  one.  However,  for  all 
of  these  elections  the  exact  margin  and  upper  bound  are 
almost  identical  to  the  Condorcet  margin.  Thus  in  most 
real  IRV  elections,  we  suspect  that  the  reported  winner  is 
likely  to  be  the  Condorcet  winner  and  the  margin  is  the 
Condorcet  margin. 

Many  of  the  elections  studied  have  margins  which  are 
quite  large.  This  suggests  that  these  elections  should  be  ef¬ 
ficiently  auditable.  However,  these  are  not,  in  many  cases, 
hotly  contested  elections.  The  2010  Oakland  School 
Board  Director  for  District  6  was  the  only  candidate  on 
the  ballot.  It  is  unclear  if  such  large  margins  would  re¬ 
main  in  larger,  more  contested  elections  such  as  state 
elections. 


6  Conclusions 

Alternatives  to  plurality  voting  like  instant-runoff  vot¬ 
ing,  if  deployed,  should  be  accompanied  by  risk-limiting 
audits.  In  this  paper,  we  have  initiated  the  study  of 
risk-limiting  audit  procedures  for  these  alternative  single¬ 
winner  election  systems 

We  classify  the  alternative  voting  systems  we  study  into 
two  groups:  scored  and  ranked  systems.  Scored  voting 
systems  can  be  audited  using  the  analysis  and  algorithms 
from  plurality  election  auditing.  However,  some  ranked 
systems  require  a  different  approach.  Condorcet  elections 
where  there  is  a  Condorcet  winner  can  be  converted  into 
a  multiple-context  plurality  election.  For  instant-runoff 
voting,  the  question  appears  to  be  more  complicated. 

We  propose  three  methods  for  auditing  IRV  elections. 
Auditing  the  taint  requires  an  efficient  method  of  calculat¬ 
ing  the  margin  (or  a  lower  bound)  and  a  way  to  compute 
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how  much  an  error  in  recording  a  ballot  affects  the  mar¬ 
gin —  the  ballot’s  taint —  an  open  problem.  Auditing  the 
elimination  order  constructs  a  set  of  multi-winner  plural¬ 
ity  elections  based  on  an  optimal  choice  of  elimination 
sets  used  to  eliminate  candidates  in  each  round  and  audits 
each  of  those.  Auditing  by  error  detection  is  similar  to 
auditing  the  taint  except  that  all  errors  are  assumed  to  be 
as  bad  as  possible  and  modify  the  margin  by  2. 

Except  for  auditing  by  error  detection  —  which  is  es¬ 
sentially  the  original  ballot-based  method  for  auditing 
plurality  election  —  knowledge  of  the  margin  for  an  IRV 
election  is  not  sufficient  for  auditing. 

An  important  question  is  whether  the  effects  seen  in 
the  relatively  small  elections  for  which  we  have  data  will 
be  present  in  larger  state  or  national  elections.  Obtaining 
data  from  such  elections  (or  polling  data)  could  be  quite 
valuable.  As  with  plurality  elections,  there  is  a  dearth  of 
information  on  the  nature  and  distribution  of  real  ballot 
errors.  Such  data  could  be  used  to  optimize  the  statistical 
efficiency  of  an  auditing  procedure. 

We  do  not  believe  that  our  proposed  auditing  proce¬ 
dures  are  the  last  word  on  risk-limiting  audits  of  alter¬ 
native  election  systems;  we  hope  that  future  work  will 
provide  simpler  and  more  efficient  audits.  As  things 
stand  now,  however,  some  voting  systems  (e.g.,  Borda 
counts)  appear  to  be  substantially  easier  to  audit  than  oth¬ 
ers  (e.g.,  IRV).  We  believe  that  ease  of  auditing  should 
be  a  criterion  when  voting  systems  are  considered  for 
adoption. 
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A  IRV  tabulation  algorithm 

One  way  to  perform  IRV  tabulation  is  described  in  Algo¬ 
rithm  4.  It  takes  as  input  the  set  of  candidates  A,  the  max¬ 
imum  number  of  rounds  of  the  algorithm  to  perform  p, 
and  the  set  of  ballots  {y,  £  n(A)}"=1.  It  iteratively  elim¬ 
inates  the  candidates  with  the  fewest  top  choice  votes  — 
y ,-  ( 1 )  is  the  top,  noneliminated  choice  on  ballot  i,  and  then 
removes  the  candidates  from  every  ballot  on  which  they 
appear.  As  output,  it  produces  the  winner,  if  any  after 
p  rounds,  the  set  of  candidates  eliminated  in  each  round, 
and  the  modified  set  of  ballots  after  candidates  have  been 
eliminated. 

As  discussed  in  Section  4.2,  there  several  rules  for 
choosing  which  candidates  to  eliminate  in  each  round. 
By  abstracting  the  choice  of  the  elimination  set,  all  va¬ 
rieties  of  IRV  can  be  described  at  once.  The  function 
EliminationSet  (B)  in  Algorithm  4  takes  a  set  of 
ballots  B  and  returns  the  set  of  candidates  to  be  elimi¬ 
nated  next.  For  example,  using  the  base  IRV  elimination 
rule,  EliminationSet  ( B )  returns  the  candidate  with 
the  fewest  top-choice  votes.  The  SF  RCV  elimination 
rule  returns  largest  set  of  candidates  E  such  that  the  sum 
of  the  top-choice  votes  for  all  candidates  in  E  is  less  than 
the  number  of  top-choice  votes  for  all  of  the  candidates 
not  in  E,  S.F.,  Cal.,  Charter  art.  XIII,  §  13.102(e) 
(2002). 

Rather  than  iterating  over  each  ballot  every  time,  one 
can  pick  smarter  representations  such  as  keeping  track  of 
how  many  ballots  with  each  particular  candidate  ranking 
exist  or  using  tree  data  structure  in  which  paths  from  the 
root  to  a  node  correspond  to  candidate  rankings  [24].  Us¬ 
ing  a  tree,  eliminating  a  candidate  involves  recursively 
removing  nodes  corresponding  to  that  candidate  and  merg¬ 
ing  their  children. 

B  Examples  for  IRV  margins 

In  this  appendix  we  give  some  toy  examples  of  IRV  elec¬ 
tions  that  illustrate  two  points.  First,  a  small  number  of 
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Algorithm  4:  IRV  -  tabulating  IRV  results 
inputs:  candidates  A,  rounds  p,  ballots  {y,-  £  II(A)}"=1 
outputs:  Winner,  ElimOrder,  modified  {y,} 

Winner  4-  0 
ElimOrder  <-() 

r  ■ f-  0 

while  r  <  p  do 

r  £-  r+  1 
foreach  c  £  A  do 

e(c)^£;-Lii(y/(i)=c) 
if  maxe{g(c)}  >  5  £c Q(c)  then 
Winner  +-  argma  x£,{g(c)} 

Append  A  \  {Winner}  to  ElimOrder 

break 

else 

E  ■£-  EliminationSet  ( {y,} ) 

Append  E  to  ElimOrder 

A^A\E 

foreach  y,  do 

yi^-yt\E 


errors  can  dramatically  change  the  outcome  of  an  IRV 
election.  Secondly,  the  IRV  margin  can  be  smaller  than 
the  Condorcet  margin,  even  when  IRV  elects  the  Con- 
dorcet  winner. 

B.l  IRV  can  be  sensitive  to  small  errors 

IRV  is  sensitive  to  errors,  in  the  following  sense:  Switch¬ 
ing  even  a  single  vote  from  one  losing  candidate  to  an¬ 
other  (or  fabricating  a  vote  for  a  losing  candidate)  may 
be  enough  to  change  the  winner  of  an  election.9  We 
illustrate  this  via  a  simple  example.  Consider  the  six  can¬ 
didate,  1000  ballot  election  in  Table  3.  Zoe  has  the  fewest 
votes  of  any  candidate.  She  is  eliminated  in  the  first  round 
and  ultimately  Velma  wins  with  496  votes.  Ulric  comes 
in  second  with  379  votes.  Naively,  one  might  say  that 
Velma  won  with  a  margin  greater  than  10%  (either  1 1.7% 
or  about  13.4%  depending  on  whether  the  denominator  is 
1000  or  379+496  =  876). 

If  an  adversary  is  able  to  arrange  for  a  single  Y  X  V 
ballot  to  be  counted  as  a  Z  Y  ballot,  then  we  get  the 
election  in  Table  4.  Here,  the  small  error  cascades  through 
the  rest  of  the  rounds  and  Ulric,  who  previously  came 
in  second,  is  the  winner  with  379  votes.  The  correct 
winner,  Velma,  does  not  even  make  it  to  the  final  round. 
Instead,  Xavier,  who  was  previously  eliminated  in  the 
second  round  makes  it  all  the  way  to  the  final  round  to 

9I.  D.  Hill  describes  a  slightly  different  example  of  instability  in 
a  real  Single  Transferable  Vote  election  —  the  multiseat  analogue  of 
instant-runoff  voting.  Hill  points  out  that  a  change  in  a  single  ballot’s 
15th  choice  (out  of  23)  would  result  in  a  different  winner.  In  this  case,  it 
was  the  difference  between  voting  for  one  of  the  (eventual)  winners  and 
the  closest  runner  up  rather  than  between  two  losers  [18]. 


lose  with  295  votes.  Again,  naively,  the  margin  appears  to 
be  quite  large  (either  8.4%  or  about  12.5%).  This  example 
shows  that  intuition  about  margin  calculations  in  plurality 
elections  may  not  be  applicable  to  IRV  elections. 

B.2  Margins  for  Condorcet  versus  IRV 

The  margin  for  IRV  may  be  smaller  than  the  Condorcet 
margin.  Consider  an  election  between  Xavier,  Yolanda, 
and  Zoe.  Only  36  ballots  were  cast  in  this  election,  and 
the  results  are  summarized  in  Table  5. 

Under  IRV,  in  the  first  round  Xavier  gets  1 1  votes, 
Yolanda  15  votes,  and  Zoe  10  votes,  so  Zoe  is  eliminated. 
However,  the  supporters  of  Zoe  break  unanimously  for 
Xavier  over  Yolanda,  so  in  the  final  round  Xavier  defeats 
Yolanda  21  votes  to  15  and  Xavier  is  the  IRV  winner.  The 
simple  lower  bound  for  the  margin  of  this  election  is  one 
vote,  the  gap  between  Xavier  and  Zoe  in  the  first  round. 
Note  that  Xavier  is  also  the  Condorcet  winner  of  this 
election  —  voters  prefer  Xavier  to  both  Yolanda  and  Zoe 
by  21  to  15.  The  Condorcet  margin  is  therefore  six  votes. 
Further,  voters  also  prefer  Yolanda  to  Zoe  21  to  15  so  the 
minimum  difference  in  preference  between  candidates  is 
also  six.  However,  the  IRV  margin  really  is  two  votes 
since  one  ballot  shifted  from  Xavier  to  Zoe  will  cause 
Xavier  to  be  eliminated  in  the  first  round  and  Yolanda  to 
win. 
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Table  3:  Unmodified  six  candidate,  1000  ballot  IRV  election. 


Candidate 

Round  1 

Round  2 

Round  3 

Round  4 

Final 

Ulric 

199:  U 

199:  U 

199:  U 

199:  U 

199:  U 

180:  WU 

180:  WU 

Velma 

200:  V 

200:  V 

200:  V 

200:  V 

200:  V 

170:  X  V 

170:  X  V 

170:  X  V 

126:  XX  V 

Wilard 

180:  WU 

180:  WU 

180:  WU 

— 

— 

Xavier 

170:  X  V 

170:  X  V 

— 

— 

— 

Yolanda 

126:  Y  X  V 

126:  Y  X  V 

126:  Y  X  V 

126:  Y  X  V 

— 

125:  Z  Y 

125:  Z  Y 

125:  Z  Y 

Zoe 

125:  Z  Y 

— 

— 

— 

— 

Table  4:  IRV  election  in  Table  3  with  a  single  Y  X  V  ballot  changed  to  Z  Y. 


Candidate 

Round  1 

Round  2 

Round  3 

Round  4 

Final 

Ulric 

199:  U 

199:  U 

199:  U 

199:  U 

199:  U 

180:  WU 

180:  WU 

Velma 

200:  V 

200:  V 

200:  V 

200:  V 

— 

Wilard 

180:  WU 

180:  WU 

180:  WU 

— 

— 

Xavier 

170:  X  V 

170:  X  V 

170:  X  V 

170:  X  V 

170:  X  X 

125:  X  X  V 

125: /X  V 

125: /X  V 

125:  X  X  X 

Yolanda 

125:  Y  X  V 

— 

— 

— 

Zoe 

126:  Z  Y 

126:  Z  X 

— 

— 

— 

Table  5:  IRV  election  where  the  IRV  margin  is  smaller  than  the  Condorcet  margin. 


Candidate 

Round  1 

Final 

Xavier 

6:  X  Y  Z 

6:XY/ 

5:  X  Z  Y 

5:  X  Z  Y 

10:  Z  X  Y 

Yolanda 

10:  Y  X  Z 

10:  Y  X  Z 

5:  Y  Z  X 

5:  Y  Z  X 

Zoe 

10:  Z  X  Y 

— 
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